Method of managing proxy objects

ABSTRACT

A method of managing proxy objects within CAD Models by attaching Meta Data to each Proxy and HD Object and translating 2D coordinates into 3D coordinates from within a 3D CAD model with additional data being added through a 360 viewer. The method enables the user to programmatically swap one Proxy Object with one or more HD Objects. All Proxy Objects and HD Objects are stored in a secure database structure while providing access by users to the proxy objects and all related product information. Non-technical and non-CAD users can configure objects within a space by selecting an object, browsing a catalog of possible alternative objects, viewing specific product details and then selecting the object to replace the selected object. Once a new object is selected, a photo realistic 360 image of a scene is created in real time.

PRIORITY APPLICATION

In accordance with 37 C.F.R. § 1.76 a claim of priority is included inan Application Data Sheet filed concurrently herewith. Accordingly, thepresent invention is a continuation of U.S. patent application Ser. No.17/135,610, entitled “METHOD OF MANAGING PROXY OBJECTS” filed Dec. 28,2020, which is a continuation of U.S. patent application Ser. No.16/813,282, entitled “METHOD OF MANAGING PROXY OBJECTS” filed Mar. 9,2020 and issued as U.S. Pat. No. 10,878,138, issued Dec. 29, 2020, whichis a continuation of U.S. patent application Ser. No. 16/585,048,entitled “METHOD OF MANAGING PROXY OBJECTS” filed Sep. 27, 2019 andissued as U.S. Pat. No. 10,635,841, issued Apr. 28, 2020, which iscontinuation-in-part application of U.S. patent application Ser. No.16/225,755, entitled “METHOD OF MANAGING PROXY OBJECTS”, filed Dec. 19,2018 and issued as U.S. Pat. No. 10,430,997, issued Oct. 1, 2019, whichis a continuation-in-part application of U.S. patent application Ser.No. 15/903,758, entitled “METHOD OF MANAGING PROXY OBJECTS”, filed Feb.23, 2018 and issued as U.S. Pat. No. 10,198,863, issued Feb. 5, 2019,which claims priority to U.S. Provisional Patent Application No.62/462,665 entitled “METHOD OF MANAGING PROXY OBJECTS,” filed Feb. 23,2017. The contents of the above referenced applications are incorporatedherein by reference.

FIELD OF THE INVENTION

The present invention concerns computer generated images and relatedproduct data. In particular, the present invention concerns improvingthe process of managing objects within CAD Models and generatingcomputer generated images with related product data which is availableas a 3D Specification.

BACKGROUND OF THE INVENTION

Computer Aided Design (“CAD”) software has been widely used to createvirtual models of buildings and products for decades. In addition,Computer Generated Images (CGI) have been used to create virtual imagesfrom video games to movies and virtual worlds. The existing processesare cumbersome, creating large files which are problematic to share andmove from user to user and require a tremendous amount of storage andcomputer power to process into photo realistic renderings. In addition,the CAD files and CGI files have competing needs, which makes itdifficult for users to efficiently create photo realistic images withouttremendous computing power. The CAD files, ideally, need to be small tobe efficient for users to save and share; while the CGI files need richdata to be able to create photo realistic images. Attempts have beenmade to address the file size issue by introducing Proxy Objects intothe CAD model. Proxy Objects are light weight objects with limiteddetail and smaller file size as compared to large weight HD Objects withmaximum detail and larger file size.

Current solutions link a single Proxy Object to a single HD Object whichare memory intensive as they have all the details required to create aphoto realistic image once rendered. Advancements in computer hardwareare decreasing the time to produce photo realistic images. While thecurrent solutions help address the file size issue, they requireindividual users to manually create a Proxy Object for each HD Object.If a user wants to change the HD Object, the user is required to changethe Proxy Object manually within the CAD model and then re-render thescene which requires the user to have specific software, hardware andspecific technical skills.

Many manufacturers have made their HD CAD Objects available publicly tohelp CAD users include their products in CAD Models. However, once auser has the manufacturer's HD Object, they have all the details of howthe product is designed. However, in doing so, the manufacturers aregiving their intellectual property to users without regard to how thepublic may use their information. This is very similar to the early daysof music being made available online where users were able to copy andshare music with little or no control by the music's owner. Users copiedand shared music freely, creating a major problem for music owners.

The current solutions are not practical for many applications. Forexample, a user who wants to create a CAD model of a single-family homeand show alternative appliances, cabinets, countertops, backsplashes,light fixtures, flooring and paint colors could easily have more than400 million combinations. This is not practical and is the main reasonthat home builders have not been able to virtually model alternativeselections for consumers with photo quality rendering. Similarscalability issues exist for commercial designers and architects whoneed to model multiple combinations of selections and retailers who mustrely on actual photography to represent their products on line.

While Proxy Objects have existed for years and make the CAD models moreefficient, all Proxy Objects have been associated only with a single HDObject in a one-to-one relationship. This limits the ability to scaleand requires a CAD user to modify the CAD model each time they want tomodify the output of the CAD file. What is lacking in the industry is amethod of managing light weight Proxy Objects and connecting a genericproxy object to one or more HD Objects using Meta Data.

Currently the 3D objects in CAD Models are disconnected from the productinformation. Users construct buildings using 2D paper plans with 2Dspecifications. Matching the details on the plans with thespecifications details is cumbersome and time consuming. BIM modelsoffered the promise of solving much of these issues by creating 3DModels of buildings with content information. However, this shifted theburden from the contractors matching manual information to thearchitects building this information within BIM models and mostarchitects don't take the time to add the product information within aBIM. In addition, a BIM requires specialized software, hardware andtraining to use.

The Product Data is not connected digitally with the 3D objects andthere is no system in place to manage the Product Lifecycle—meaning thatend users don't have any way to know if a product is still available orif it will be discontinued at some future date. Additionally, once amanufacturer discontinues a product, they typically delete allreferences to that product on their website. This makes it verydifficult for end users to research product information on products thathave been installed.

Current solutions do not provide any way for a Non-CAD user to commentor add data to a 3D Model from outside of CAD Software. This requires aCAD User to navigate with a Non-CAD user through the CAD Software tomake changes to a CAD Model or, more frequently, the CAD User prints a2D set of plans for the Non-CAD User to add comments and additionalinformation on paper. The CAD User then takes the informationcommunicated via paper and updates the CAD Model. This is a very manualprocess that is time consuming and creates many opportunities forerrors.

SUMMARY OF THE INVENTION

Disclosed is a method of managing proxy objects within CAD Models by anapplication executing on a computer device for attaching Meta Data toeach Proxy and HD Object. The Meta Data enables the user toprogrammatically swap one Proxy Object with one or more HD Objects. AllProxy Objects and HD Objects are stored in a secure database and relatedvia Meta Data.

An objective of the invention is to swap various Proxy Objects—each forone or more HD Objects.

Another objective of the invention is to prevent the complete disclosureof manufacturer's HD Object CAD files by providing Proxy Objects forScene creation while protecting the HD Object in a Data Vault.

Another objective of the invention is to connect manufacturers' productdata (including images, videos, product information and relateddocuments provided by a manufacturer) with each HD Object to create a 3DSpecification for users to access visually directly from within a CADModel or a Scene.

Another objective of the invention is to reduce the storage spacerequired for displaying virtual and interchangeable models used tocreate photo realistic renderings. The use of Proxy Objects alsoreducing the need for high speed internet access and computer processorsused in the substitution of Proxy Objects for HD Objects.

Another objective of the invention is to enable non-technical users toswap HD Objects within a 360 view and obtain updated 360 photorealistic, renderings with 3D Specifications in the 360 View. Users withthe correct permissions can view the product history which includesprevious 3D objects and associated product data.

Another objective of the invention is to enable non-technical user tointeract with a 3D CAD Model through 360 Viewer Software to addcomments, request changes to the 3D Model, add additional Product Dataor to add photos documented field conditions during construction.

Other objectives and further advantages and benefits associated withthis invention will be apparent to those skilled in the art from thedescription, examples and claims which follow.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawings will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 is a summary flow chart of the ecosystem used in this invention.

FIG. 2 is a pictorial illustrating the protecting content owner'sintellectual property.

FIG. 3 is a flow chart illustrating setting up a new company and users.

FIG. 4 is a flow chart illustrating the process for creating new 3Dobjects.

FIG. 5 is a flow chart illustrating the process for adding product datato the Product Database.

FIG. 6 is a flow chart illustrating the process for adding Proxy Objectsto a CAD Model.

FIG. 7 is a flow chart illustrating the process for swapping Objects ina CAD Model.

FIG. 8 is a flow chart illustrating the process for swapping Objects in360 Interactive Views Over the Internet.

FIG. 9 is a pictorial illustrating the relationship between One ProxyObject and Many HD Objects.

FIG. 10 is a pictorial illustrating the process for swapping Objects inthe 360 Viewer Software.

FIG. 11 is a pictorial illustrating Swapping an Object in a CAD Model.

FIG. 12 is a chart illustrating the Meta Data Relationship between ProxyObjects and HD Objects.

FIG. 13 is a flowchart illustrating the process for automaticallygenerating a 360 tour from a CAD Model

FIG. 14 is an example of dynamically managing menu structure for 3DObjects and Product Data.

FIG. 15 is an example of a 3D Object and Product Data Assembly.

FIG. 16 is an example of Bi-Directional Data Flow between the 360 ViewerSoftware, a 3rd Party database and a CAD Model.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Detailed embodiments of the instant invention are disclosed herein;however, it is to be understood that the disclosed embodiments aremerely exemplary of the invention, which maybe embodied in variousforms. Therefore, specific functional and structural details disclosedherein are not to be interpreted as limiting, but merely as a basis forthe claims and as a representation basis for teaching one skilled in theart to variously employ the present invention in virtually anyappropriately detailed structure.

Set forth is the method of managing proxy objects within CAD Models byattaching Meta Data to each Proxy and HD Object. The Meta Data enablesus to programmatically swap one Proxy Object with many possible HDObjects. All Proxy Objects and HD Objects are stored in a database andrelated via the Meta Data. The HD Objects provided by productmanufacturers contain intellectual property “IP”. Manufacturers canupload the HD Objects, related Meta Data and Product Data to a DataVault programmatically or through a graphical user interface over theInternet or they can connect to the invention via an API developed byinventor to connect product stored in a manufacturer's database with the3D Proxy and HD Objects. The System connects each HD Object to theappropriate Proxy Object using the Meta Data (see FIG. 12), and protectsthe HD Object CAD information in the Data Vault (see FIG. 2). The DataVault is only accessible by each manufacturer's authorized users and acontroller's software to protect each manufacturer's HD Objects in muchthe same way music services like iTunes protect music owners' content.End users never have access to the manufacturers' HD CAD Objects.

As an example, FIG. 4 illustrates the process for creating 3D Objects.Once an Object is released for use, a CAD User can add a Proxy Objectfor a faucet in a CAD model by browsing the product catalog (see FIG.6). The Proxy Object is only a rough representation of an actual faucetoffered by a manufacturer. The Meta Data connects a Proxy Object toone-or-more HD Objects (see FIG. 9). Using Meta Data, a CAD User or aNon-CAD User can swap the Proxy Object for a specific HD Object byselecting a specific Object from the Product Catalog (see FIG. 7 for CADModel process and FIG. 8 for non-CAD Model process). The System providesa visual representation of the HD Object selected but maintains theProxy Object in the CAD Model to protect the IP while enabling the userto view 3D Specification. The CAD User can render the scene using 3rdParty rendering software at which time the system switches the ProxyObject for the correct HD Object and provides a photo realistic, imageof the scene with the HD Object (see FIG. 11). For Non-CAD Users, theScene is presented through a web-browser or mobile app. The Non-CAD usercan select an Object, browse a product catalog to swap the object andselect the object. Then the System automatically switches the HD Objectwith the one selected by opening the CAD Model swapping the Objects,rendering the new Scene and then displays the Scene with the new Objectand corresponding 3D Specification (see FIG. 10).

Referring to FIG. 1 the product database stores all product informationincluding, but not limited to product specifications, productattributes, product images, product installation guides, product videosand product dimensions. The product database contains data from allmanufacturers that are in the network. Manufacturers can use the ProductData API developed as part of this invention to connect productinformation from within their own database(s) with the 3D Objectdatabase. The data within the product database or connected via theProduct Database API is managed through the business logic layer and isconverted to 3D Specifications for viewing by Users at the time theyselect an Object whether in the CAD Model or through the 360 viewersoftware.

The 3D objects database stores all the product CAD models which includeproxy objects and HD objects. Each object contains meta data thatconnects the proxy objects to the related product content. Meta dataconnects the product information with the 3D objects and is also used tosort and filter data for end users. The business logic is theapplication layer to the invention and it manages the business unithierarchy, product specifications, the product lifecycle, versioncontrol, specifying products for use within the product hierarchy andspecific CAD models and swapping proxy objects.

The 360 visualization software takes 360 panoramic photographic or 360panoramic virtual images and convert them into viewable images in the360 viewer software. In addition, this software converts 3D coordinatesfrom within the CAD model into 20 coordinates which enables thevisualization software to automatically place icons for movement fromone camera location to another and to automatically places icons thatenable users to select objects to view product information or swapobjects based upon the business logic. The 360 Viewer Software presentsthe 360 panoramic images to end users through a web browser or mobileapp. The viewer software enables users to move from one panoramic imageto another (as if they are walking through a space) and to selectobjects to view product information known as a 3D Specification or swapwith other objects in what is referred to as a virtual tour.

CAD users access the invention through a plugin added to a 3rd party CADsoftware. These users can create or select proxy objects from within the3D object database, view product data for HD objects and swap objectsfrom the 3D object database directly within a CAD model. Non-CAD usersaccess the invention through a website or mobile app using the 360viewer software. These users can view panoramic images and move from onepanoramic image to another as if they are walking through a space. inaddition, non-CAD users are able to select objects (virtual images orphotographic images) from within the 360 viewer software to view productinformation, select optional items or swap objects or colors based onconfiguration settings established in the business logic. CAD(computer-aided design) software is 3rd party software that is used byarchitects, engineers, drafters, artists, and others to create precisiondrawings or technical illustrations. CAD software can be used to createtwo-dimensional (2-D) drawings or three-dimensional (3-D) models ofobjects or buildings. The invention has software integration with 3rdparty CAD software to facilitate the process of developing, managing andswapping proxy objects along with managing all related product data(including product information, installation guides and videos, warrantyinformation, safety data sheets and any other information that amanufacturer or users chose to associate with the 3D object). Theproduct data can be attached to any object in a CAD model or any objectrepresented in a photographic image. The product data can be changeddynamically in the database and automatically updated in the visualrepresentation.

A user, with appropriate permissions, can manage the Product Lifecycleby updating the Product Data directly in the database. Once the updatedProduct Data is approved as defined in the manufacturer's workflow, theupdated Product Data is automatically updated in the CAD Models and 360Viewer. The system retains all changes to both the 3D Objects and allProduct Data enabling users, with appropriate permissions, to view priorversions of a 3D Object or Product Data. The user can attach aneffective date when the new 3D Object or Product Data will becomeeffective or will become obsolete. Additionally, a user with theappropriate permissions can identify a different 3D Object or ProductData that will replace a current 3D Object or Product Data.

Rendering software is 3rd party software that is used by architects,engineers, drafters, artists, and others to create photo realisticimages from CAD models or objects and buildings. The rendering softwarecan produce images in real time or at specific times as requested by theuser. The invention has software integration with 3rd party renderingsoftware to facilitate the process of creating photo realistic imagesfor use in the 360 viewer software,

Referring to FIG. 2 the 3D objects database stores all the product CADmodels which include proxy objects and HD objects. Each object containsmeta data that connects the proxy objects to the related productcontent. Proxy objects are light weight representation of a HD object.is uses fewer polygons to represent the object making the file sizesmaller than the HD object. Proxy objects allow CAD users to place ageneric representation of a HD object in a CAD model. Since a proxyobject is only a representation of a HD object the CAD user does nothave enough information within the object to copy the content owner'sIP. The firewall segregates the proxy objects from the HD objects toprotect the content owner's IP through a data vault. The HD objects areheld behind the firewall in the data vault and are only accessible bythe content owner and their designated users while the proxy objects aremade available to all users. HD Objects are detailed CAD objects thatcontain technical details on how the object is constructed, detaileddimensions, materials, textures and all supporting data needed toproduce a photo realistic image of an object. Meta data connects theproduct information with the 3D objects and is also used to sort andfilter data for end users. The meta data is used to connect the proxyobjects with the HD objects across the firewall while protecting the HDobjects in the data vault.

Referring to FIG. 3 a company set up is managed within the business unitlogic and manages information for a specific company. The business unithierarchy is defined for each company and establishes the relationshipswithin a company. For example, a company may be structuredgeographically into continents, countries, regions, divisions andproducts or a company may be structured around products and thengeographically. The business unit hierarchy provides the flexibility tostructure data and user access based on each individual company's needs.If the company does not produce products that will be included in theproduct database, the user can skip that step in the setup of a newcompany. The product hierarchy works very similar to the business unithierarchy but manages product information into categories,sub-categories, sub-sub-categories—all of which are based on the productmeta data. This structure assists with sorting and filtering of datawithin the product catalog, Each manufacturer's product data is mappedto the product hierarchy within the invention. The product hierarchy isdynamic and can be modified at any time by users with the appropriateuser permissions making it easy For a user to view the product data in ahierarchy view that works best for each individual user. FIG. 14,illustrates two examples of how a user can use the Meta Data tostructure the view of Product Data into a different hierarchy to meet auser's specific needs. The folder structure is dynamically created basedon the user's requirements and is simply a view into the data and nothard wired. This enables each user to customize their view of the datawithout compromising the overall structure of the data or other users'views of the same data. The business logic manages the process forcreating new users and user permissions. Users are assigned to a levelwithin the business unit hierarchy and their permissions within thesystem are assigned by user groups which specify what information userscan access and whether they have rights to simply view the data, editdata or add/delete data. Once a new company is created and the users arecreated, the system emails credentials to each new user.

Referring to FIG. 4 a CAD user, with appropriate permissions, can createa HD Object using the 3rd party CAD software with the invention'ssoftware plugin. The CAD user creates a detailed CAD model for theobject to accurately represent the real objects in all respects. Oncethe HD object is created, the CAD user can create the proxy object withall the required data (including materials, textures and IES lightinginformation if applicable). In addition, the CAD user can add theapplicable meta data to the proxy object. Once the HD Object and proxyobject are complete, the CAD user exports the objects to the 3D Objectdatabase. When the objects are uploaded to the 3D content database, thesystem automatically connects the objects to the relevant product datain the product database or to a manufacturer's remote database throughthe meta data. The invention includes a workflow management feature tomanage the approval of new or modified content in the 3D and productdatabases. In addition, the user can create Assemblies which groupmultiple 3D Objects into a single group or “Assembly”. This allows a CADuser or non-CAD User to view product information at the Assembly levelor drill down into the individual 3D objects to view each 3D Objectsspecific product data. FIG. 15 provides an example of an Assembly whichis a collection of 3D Objects that are grouped together. The Assemblycombines related 3D Objects with their associated Product Data andenables a user to create Meta Data and Product Data for the Assembly inaddition to the Meta Data and Product Data associated with each 3DObject. Upon review and approval of the new or modified content, theuser (with appropriate permissions) can release the content for publicconsumption by CAD users and non-CAD users.

Referring to FIG. 5, users with the appropriate permissions must loginto the systems website before adding product data to the productdatabase. When the user logs into the website, the business logicvalidates the user's credentials before allowing the user to moveforward. The user can select to enter product data one item at a timedirectly into a data screen on the website or they may select to uploaddata in bulk. If the user selects to upload data, the system (based onthe company settings) will either initiate an automated process throughthe Product Data API with the user's company system or allow the user toupload data through a pre-formatted spreadsheet. In the case ofuploading data, the system will prompt the user to browse to the filelocation for the pre-formatted spreadsheet containing the data to beuploaded. Once the user has selected the appropriate file to upload,they will click “ok”. The system will also prompt the user to browse tothe file location for the supporting documents and images to beuploaded. Once the user has selected the appropriate folder, the userwill click “ok”. If the user selects to manually enter data for aproduct, the system will display all product fields that can be entered.The user will enter all relevant product information and then click“ok”. The user will then be prompted to browse to the location of thesupporting product images and content. After selecting the content toupload, the user will click “ok”. The system will validate all dataentered by the user meets the required criteria and then the user willbe prompted to review the information prior to clicking “Add toDatabase”. Once the user has clicked add to database, the system willupdate the database with the information provided via the upload ormanual process and will add the product images and content to thedatabase. The invention includes functionality to manage the ProductLifecycle by tracking all changes to 3D Objects and Product Data. Theinvention includes a workflow management feature to manage the approvalof new or modified content in the 3D and product databases. Upon reviewand approval of the new or modified content, the user (with appropriatepermissions) can release the content for public consumption by CAD usersand non-CAD users.

Referring to FIG. 6, the CAD user starts the CAD software and loads theappropriate CAD model. The CAD user logs into the database where thesystem validates the user's permissions. The user can then open theproduct catalog from within the CAD software. The user can browse,search and filter on meta data to find the desired object to add to theCAD model. The user can decide if they want to see the product data orsimply select the product based on the summary information provide onthe initial screen of the catalog for an object. If the user clicks onthe object, they will see the product data and any associated images andproduct content in the system which is viewed as a 3D Specification,Once the user has found the object they want, they simply select theobject and then place within the model.

Referring to FIG. 7, the CAD user starts the CAD software and opens thedesired CAD model. The CAD user logs into the database where the systemvalidates the user's permissions. The CAD user selects the object withinthe CAD model that they want to swap and then click on the menu item toswap object. The system will launch the product catalog and take theuser to the correct product category and sub-category based on the metadata of the current object in the CAD Model. The user is able to browse,search and filter on meta data to find the desired object to replace inthe CAD model. The user can decide if they want to see the product dataor simply select the product based on the summary information provide onthe initial screen of the catalog for an object. If the user clicks onthe object, they will see the product data and any associated images andproduct content in the system which is viewed as a 3D Specification. TheCAD user selects the object to replace in the CAD Model and the systemthen swaps the original object with the new object along with all metadata.

Referring now to FIG. 8 the system generates interactive 360 viewsthrough the 360 viewer software for end users to interact with virtualor photographic 360 images. All users can select objects from within the360 viewer software that contain product information and/or have theoption to swap the standard object for an optional object. A user simplytaps or clicks on the object they want to swap. The user can browse theproduct catalog once they have selected the object they want to swap.The system will launch the product catalog and take the user to thecorrect product category and sub-category based on the meta data of theobject selected by the user. The business logic contains rules thatmanage mutually exclusive objects, object packages (one object changemay require the change of additional objects) and other rules thatensure a user is configuring an object properly. The user is able tobrowse, search and filter on meta data to find the desired object toreplace in the interactive viewer. If the user clicks on the object,they will see the product data and any associated images and productcontent in the system. if the user clicks on the object, they will seethe product data and any associated images and product content in thesystem which is viewed as a 3D Specification. The user selects theobject to replace in the CAD Model and the system then swaps theoriginal object with the new object along with all meta data. The CADModel related to the view is found in the 3D Content database. TheSystem will check to see if the Object(s) requested have been renderedpreviously and, if so, display the previously rendered Scene. If not,the 360 View Meta Data is used to retrieve the specific camera in theCAD Model. The spherical coordinates of the desired Object in the vieware used to retrieve the 3D coordinates in the CAD Model. When theobject is localized, it is swapped with the object the User hasselected. The SDK of the 3rd party CAD software is used so there is noneed to open the file to modify its content. Once the Object has beenswapped in the CAD Model, the Scene is rendered. Through the SDK of the3rd party rendering software, the System interprets the information thatneeds to be rendered and sends that information to the 3rd partyrendering software to be rendered. While the system is rendering, theUser can navigate through the 360 scenes and a buffer bar shows thepercentage of time left for the rendering. When the new 360 renderedScene is ready, the system will update the Scene for the User. TheSystem stores previously rendered scenes to enable the User to go backto a previous configuration without having to render the scene again.The Product Data is attached in the new Object from the Product Contentdatabase in the form of a 3D Specification and is available for theUser.

Referring to FIG. 9 a Proxy Object is a representation of a product andis not a purchasable item. The Proxy Meta Data is generic and notspecific to a single manufacturer or product. This enables the CAD Userto simply place the kitchen faucet proxy object in a CAD Model in thecorrect location. The Proxy Object also contains Meta Data whichconnects the Proxy Object to one or more HD Objects. By way of example,the Meta Data manages the relationship between Proxy Objects and HDObjects. The Meta Data structures data through the product hierarchyinto product category, sub-category and sub-sub-category. As an exampleFIG. 9 image 9.1 shows a Proxy Object of a faucet. The associated MetaData for this Proxy Object is Category—Plumbing; Sub-Category—Faucets;Sub-Sub-Category—Kitchen. Of note, the Meta Data associated with theProxy Object is generic and only serves to categorize the Object in theproduct hierarchy.

A product information example is also considered Meta Data but isspecific to a manufacturer and a specific product. The ProductInformation is connected to a product that can be purchased and containsinformation like but not limited to; manufacturer, product name, SKU,description, finish and style. The Product Information is converted intoa 3D Specification. Each HD Object is connected to the appropriateProduct Information and to the Product Hierarchy through Meta Data. Thisconnection enables the system to match a single Proxy Object withMultiple HD Objects and allows a single HD Object to be connected to theappropriate Product Information.

Referring to FIG. 10, an example of swapping an object in the 360 viewersoftware is depicted. The User selects an Object within the 360 ViewerSoftware that they want to swap. The illustration on FIG. 10 shows akitchen faucet selected. Once the User selects the Object, the Systemshows the alternative Objects available to swap. The User selects theObject to replace the Object selected or can view detailed productinformation. When the User selects a specific. Object from the ProductCatalog, the System will display detailed product information in theform of a 3D Specification.

Referring to FIG. 11, swapping object in CAD model example with a modelkitchen scene is when User selects the object within the CAD Model toswap and navigates through the product catalog to find the HD Objectthey want to swap to. The System loads the Proxy Object. Once the Systemhas replaced the generic Proxy Object with a specific Object, the Systemcan send the Scene to the 3rd Party rendering software to render. FIG.11.2 shows an example of a rendered Scene. The CAD User can continue toswap objects in the CAD Model and see the results of each selection whenrendered by the 3rd Party rendering software. FIG. 11.3 shows an exampleof a 2nd Object rendered in the kitchen scene.

Referring to FIG. 12, the relationship between HD Objects and ProxyObjects is illustrated. The Meta Data for the Product Hierarchy connectsthe Proxy Objects to the HD Objects. FIG. 12 illustrates some of theMeta Data captured by the System but is not a complete list of data.Additionally, the System is dynamic and supports adding additional MetaData over time.

Referring to FIG. 13, the User initiates the creation of a Virtual Tourfrom within the CAD Model by pushing a single button. The Softwarecollects the 3D Coordinates for all Objects and Camera Positions withinthe CAD Model. The Software uses the information from each Camera Poseto project the 3D Coordinates to 2D Coordinates. Software distinguishesbetween the cameras for the floor plan from the 360 cameras and appliesto appropriate calculations for each. The Software then initiates therendering process to convert the 3D CAD Model into Photo Realistic 360Images that will be used to create the Virtual Tour. Once the Renderingprocess is complete, the Software automatically generates the VirtualTour by placing navigational icons to move from Camera Pose to CameraPose and adds icons to enable the User to access all data associatedwith each Object. Upon completion of the process, the Software uploadsthe completed information to a 3rd Party database and provides the Userwith a URL link for the completed Virtual Tour.

Referring to FIG. 16, a User can view data from a 3rd Party database,interact with said data and enter additional data that will be uploadedto the 3rd Party database and entered back into the 3D CAD Modelautomatically or on demand as selected by a CAD User. More specifically,a User can select a location or an Object from within a 360 viewer. TheUser can enter data which may include, but not limited to, commentsregarding the model, questions for other Users to respond to which arealso known as “Request for Information” (RFI, request to make changes toa Model, documentation of quality or safety issues. This data can be inthe form of text typed into a prompt from within the 360 Viewer, voicemessage, photos, other images, video, additional files or links to otherdata on the Internet.

Once the User has identified a location and entered information into the360 Viewer, the Software identifies the specific 2D Coordinates of thelocation selected by the User. The Software uploads the informationentered by the User along with the 2D Coordinates to a 3rd Partydatabase. Once uploaded, the Software runs an automated process toTranslate 2D Coordinates into the corresponding 3D Coordinates fromwithin the 3D CAD Model and adds all new data and information into a 3rdParty database.

Once uploaded, the Software will automatically, or the User canmanually, update the 3D Model with the specific coordinates of theObject or location selected by the User in the 360 Viewer with a visualannotation of a pending request or comment from a User. The annotationin the 3D CAD Model includes meta data links to all information andassociated data uploaded by the User from within the 360 Viewer. TheSoftware then manages a User defined workflow to manage changes to the3D CAD Model, the flow of information to all Users that need to benotified and the final disposition of the information uploaded by theUser from within the 360 Viewer.

The CAD model contains floor plan and panoramic camera information andobject Meta Data. All objects with Meta Data will be shown in thevirtual tour. The System collects all the 3D coordinates (x, y, z) ofthe centroids of the bounding boxes of every relevant object that willappear in the Scenes within the 360 viewer software, i.e., those thatcontain metadata. The coordinates must be retrieved in the global systemof the 3D Model.

n=0 . . . N−1, n=index of object in model,

N=total number of relevant objects

Centroid: p _(n)=(p _(xn) , p _(yn) , p _(zn)),

The System collects all 3D camera positions in global coordinates (x, y,z). Each camera has a local system of coordinates defined by threevectors: right ({right arrow over (R)}), up ({right arrow over (U)}),and direction ({right arrow over (D)}), with respect to the globalsystem of coordinates. Using these vectors and the 3D camera position(c), the system programmatically calculates the roto-translation matrix(R|T) for every camera (see above). This matrix represents the extrinsicparameters and the global pose of the camera, and they will be useful toproject the 3D coordinates of an object to 2D pixel coordinates in thedomain of a specific camera.

m = 0… M − 1, m = index  of  camera  in  model, M = total  number  of  cameras${= \left( {R_{xm},R_{ym},R_{zm}} \right)},{= \left( {U_{sm},U_{ym},U_{zm}} \right)},{= \left( {D_{xm},D_{ym},D_{zm}} \right)},{c_{m} = {{\left( {c_{xm},c_{ym},c_{zm}} \right) - {\overset{\rightharpoonup}{R_{m}} \cdot c_{m}}} = {{\left( {- R_{xm}} \right)c_{xm}} + {\left( {- R_{ym}} \right)c_{ym}} + \left( {{- R_{zm}}c_{zm}} \right)}}}$$\left. R \middle| T_{m} \right. = \begin{pmatrix}R_{xm} & R_{ym} & R_{zm} & {{- \overset{\rightharpoonup}{R_{m}}} \cdot c_{m}} \\U_{xm} & U_{ym} & U_{zm} & {{- \overset{\rightharpoonup}{U_{m}}} \cdot c_{m}} \\D_{xm} & D_{ym} & D_{zm} & {{- \overset{\rightharpoonup}{D_{m}}} \cdot c_{m}} \\0 & 0 & 0 & 1\end{pmatrix}$

The CAD model contains floor plan and panoramic camera information andobject Meta Data. All objects with Meta Data will be shown in thevirtual tour. The System collects all the 3D coordinates of thecentroids of the bounding boxes of every relevant object that willappear in the Scenes within the 360 viewer software, i.e., those thatcontain metadata. The coordinates must be retrieved in the global systemof the 3D Model.

The system programmatically converts the 3D spatial coordinates of eachobject with Meta Data to 2D pixel coordinates in the render image. TheSystem differentiates between floorplan and panoramic cameras.

For floorplan cameras, the System sets the projection to parallel (focallength=infinite), since floor plan cameras are placed in a Top view, andruns a ray cast algorithm through all the panoramic cameras. If a querycamera is seen, the System can transform directly the coordinates of thecamera from 3D to 2D using the render viewport screen coordinates. Thequery point in the screen viewport matches with the query pixel in therender image.

For panoramic cameras, the System sets the field of view to 360 degreesand runs a ray cast algorithm through all the objects with Meta Data. Ifan object is seen, the System converts the 3D global coordinates of theobject (p_(n)) to the local system of the query camera (p′_(n)) usingits R|T matrix.

p′ _(n) =R|T _(m) p _(n)

Then it projects these 3D local coordinates to 2D texture coordinatesusing cube mapping (see ANNEX 1). Finally, the System converts fromcubic to equirectangular coordinates (Cartesian to polar).

When the 2D coordinates are calculated, the System starts the rendering.Floorplan cameras are rendered in parallel projection and matches theviewport size used for coordinates calculation. Panoramic cameras arerendered in perspective projection with a FOV=360°, and the result areequirectangular images that match the 2D coordinates.

The set of 2D coordinates are assigned to theft corresponding images andthe Virtual Tour is generated with icons and links to the Meta Data foreach object.

The Virtual Tour is created through the 360 visualization softwareautomatically. Once created, the System saves the resulting files andpublishes the virtual tour with a unique URL.

ANNEX 1 - XYZ to cubic UV void convert_xyz_to_cube_uv(float x, float y,float z, int *index, float *u, float *v) { float absX = fabs(x); floatabsY = fabs(y); float absZ = fabs(z); int isXPositive = x > 0 ? 1 : 0;int isYPositive = y > 0 ? 1 : 0; int isZPositive = z > 0 ? 1 : 0; floatmaxAxis, uc, vc; // POSITIVE X if (isXPositive && absX >= absY &&absX >= absZ) { // u (0 to 1) goes from +z to −z // v (0 to 1) goes from−y to +y maxAxis = absX; uc = −z; vc = y; *index = ’RIGHT’; } //NEGATIVE X if (!isXPositive && absX >= absY && absX >= absZ) { // u (0to 1) goes from −z to +z // v (0 to 1) goes from −y to +y maxAxis =absX; uc = z; vc = y; *index = ’LEFT’; } // POSITIVE Y if (isYPositive&& absY >= absX && absY >= absZ) { // u (0 to 1) goes from −x to +x // v(0 to 1) goes from +z to −z maxAxis = absY; uc = x; vc = −z; *index =’TOP’; } // NEGATIVE Y if (!isYPositive && absY >= absX && absY >= absZ){ // u (0 to 1) goes from −x to +x // v (0 to 1) goes from −z to +zmaxAxis = absY; uc = x; vc = z ; *index = ‘BOTTOM’ ; } // POSITIVE Z if(isZPositive && absZ > = absX && absZ >= absY) { // u (0 to 1) goes from−x to +x // v (0 to 1) goes from −y to +y maxAxis = absZ; uc = x; vc =y; *index = ‘FRONT’; } // NEGATIVE Z if (!isZPositive && absZ >= absX &&absZ >= absY) { // u (0 to 1) goes from +x to −x // v (0 to 1) goes from−y to +y maxAxis = absZ; uc = − x; vc = y; *index = ‘BACK’; } // Convertrange from −1 to 1 to 0 to 1 *u = 0.5f * (uc maxAxis + 1.0f) *panorama_width; *v = panorama_width − 0.5f * (vc maxAxis + 1.0f) *panorama_width; }

In one embodiment the invention is a computer driven method of managingproxy objects within CAD Models comprising the steps of: uploadingObjects by a computer device having a memory to an accessible securedatabase; attaching meta data by an application executing on thecomputer device to a 3D CAD Model and to each Object stored in saidsecure database; matching a Proxy Object by an application executing onthe computer device to at least one or more said Objects using a productcategory hierarchy; locating at least one said Proxy Object from saidsecure database; placing said located Proxy Object in said 3D CAD Model;selecting HD Objects to replace selected Proxy Objects on a scene;displaying a computer generated, computer image or photo realisticrendered image of the scene where Proxy Objects have been swapped withselected HD Objects within the scene; selecting an Object from within aviewer; identifying 2D coordinates of said location by an applicationexecuting on the computer device; entering additional data to saidlocation through said viewer; adding said additional data into thesecure database, translating said 2D coordinates into 3D coordinates byan application executing on a computer device outside of the 3D CADmodel and adding said additional data into the 3D Model at thecorresponding 2D Coordinates of said location; and displaying photorealistic rendered images of multiple scenes with HD Objects, placingselected 3D Specification icon by an application executing on thecomputer device in an appropriate position within said 3D CAD Model and3D Specifications for Users to access via said 3D CAD Model, a browseror mobile app.

TERMS DEFINED: 360 View is a panoramic view including 360° horizontalview and 180° vertical view for a virtual or photographic image. 3DSpecifications is a method of combining product information into avisual format that can be accessed directly by clicking on an Objectwithin a CAD Model or by clicking on an Object within the 360 viewersoftware.

2D Coordinates represent the X and Y coordinates of a location or Objectwithin a VR Image or Photograph.

3D Coordinates represent the X, Y and Z coordinates of a location orObject within a 3D CAD Model.

3D Specifications are a visual representation of Product Data that caninclude, but not limited to, product specifications, product photos orvideos, installation guides, care and maintenance guides, safety datasheets or any other information that can be associated with a product.

Assembly is a group of objects that grouped into a single group or“Assembly”. This allows a CAD user or non-Cad User to view productinformation at the Assembly level or drill down into the individual 3Dobjects to view each 3D Objects specific product data.

Bi-Directional Data is a method of moving data in both directionsbetween a 3D Model and the 360 Viewer.

Bounding box is a box with the smallest volume within an Object lies.Centroid is the center of mass of a Bounding Box.

CAD Model is computer file used by CAD software programs and contains 2Dor 3D graphical information for buildings and the objects withinbuildings. CAD Models are used to create the construction drawings usedto construct buildings and more recently used to create photo realisticrenderings of views within the CAD Model. CAD User is a user that isaccessing the System through a third-party CAD software.

CAD User is a User that accesses the Software from within a 3D CADModel.

Camera pose is the combination of position and orientation of the camerawithin a 3D space, relative to the global coordinate system.

Building Information Model “BIM” is a 3D CAD Model that includesinformation about a building and objects within the building.

Computer Aided Design (“CAD”) is a software platform that allows usersto create virtual models of buildings, landscapes, products, etc.

Computer Graphic Image (“CGI”) is the final output used for representinga virtual object or scene.

Data Vault is a secure data base that contains HD Objects and relatedproduct information for manufacturers, and restricts access to thisinformation to authorized users from a manufacturer. HD Object is adetailed CAD file with all supporting data needed to produce a photorealistic image of an object.

Meta Data refers to data associated with a Proxy Object or an HD Object.Meta Data is used to connect Proxy Objects to one or more HD Objects andis used to allow users to browse, sort and filter information in theProduct Catalog.

Non-CAD User is a user of the System who accesses the System withoutusing a third-party CAD software.

Object is used to include Proxy Objects and HD Objects collectively.

Plugin is specific software within the System that is installed within3rd party software.

Poly Count refers to the number of polygons used to create an objectwithin a CAD Model.

Product Catalog is the collection of all HD Objects and related productdata within the System.

Product Category Hierarchy is a data hierarchy used to categorize andmap Proxy Objects to HD Objects.

Product Data is a general term referring to all information associatedwith a specific product. The Product Data is converted into 3DSpecifications through the invention.

Product Data API is specific application programming interface toconnect a manufacturer's Product Data which is stored in a remotedatabase with the 3D Objects and Meta Data contained within theinvention.

Product Lifecycle refers to the evolution of a product's 3D Object andor Product Data over time. As an example, manufacturers routinely updatefeatures or components within a product but don't change the SKU. TheProduct Lifecycle manages the changes in the product over time andincludes effective dates for a product to be released or retired and mayinclude suggested replacement products when a product is discontinued.

Proxy Object is an object placed in a CAD Model which is used to connectan object to Product Data. A Proxy Object may include a light weightrepresentation of an object which uses fewer polygons to represent theobject, making the file smaller than the HD Object.

Remote Database is a database controlled by a third party, like amanufacturer, where the third party manages Product Data.

Render is the final visual output once the rendering process iscomplete.

Rendering is the process of applying materials, lighting, reflectionsand shading to a CAD scene to create a photo realistic image.

Scene is a view from within a CAD file which is used to create thevisual output.

System refers to all elements as depicted in FIG. 1.

Translate 2D to 3D Coordinates is the process of translating 2DCoordinates into the corresponding 3D Coordinates.

Virtual Model is a collection of scenes within a single CAD file whichcan represent a home, budding, product or environment.

User is used to refer to all Users of the System.

The terms “comprise” (and any form of comprise, such as “comprises” and“comprising”), “have” (and any form of have, such as “has” and“having”), “include” (and any form of include, such as “includes” and“including”) and “contain” (and any form of contain, such as “contains”and “containing”) are open-ended linking verbs. As a result, a method ordevice that “comprises,” “has,” “includes” or “contains” one or moresteps or elements, possesses those one or more steps or elements, but isnot limited to possessing only those one or more elements. Likewise, astep of a method or an element of a device that “comprises,” “has,”“includes” or “contains” one or more features, possesses those one ormore features, but is not limited to possessing only those one or morefeatures. Furthermore, a device or structure that is configured in acertain way is configured in at least that way, but it may also beconfigured in ways that are not listed.

One skilled in the art will readily appreciate that the presentinvention is well adapted to carry out the objectives and obtain theends and advantages mentioned, as well as those inherent therein. Theembodiments, methods, procedures and techniques described herein arepresently representative of the preferred embodiments, are intended tobe exemplary, and are not intended as limitations on the scope. Changestherein and other uses will occur to those skilled in the art which areencompassed within the spirit of the invention and are defined by thescope of the appended claims. Although the invention has been describedin connection with specific preferred embodiments, it should beunderstood that the invention, as claimed, should not be unduly limitedto such specific embodiments. Indeed, various modifications of thedescribed modes for carrying out the invention which are obvious tothose skilled in the art are intended to be within the scope of thefollowing claims.

What is claimed is:
 1. A method of managing proxy objects within CADmodels comprising: uploading objects to a computer device having amemory coupled to an accessible secure database in which the objects arestored, the objects including proxy objects and HD objects matched bymeta data attached thereto; placing at least one the proxy objects in ascene of a 3D CAD model; selecting one or more HD objects to replace theproxy object in the scene of the 3D CAD model; displaying photorealistic rendered images of the scene at specific times, wherein theproxy object in the scene of the 3D CAD model has been swapped with theselected one or more HD objects within the scene; and placing, by anapplication executing on the computer device, a selected 3Dspecification icon within the 3D CAD model for a user to access via abrowser or a mobile app.
 2. The method of managing proxy objects withinCAD models according to claim 1, further comprising matching, by theapplication executing on the computer device, the proxy object in thescene of the 3D CAD model to the selected one or more HD objects using aproduct category hierarchy.
 3. The method of managing proxy objectswithin CAD models according to claim 1, further comprising: identifying,by the application executing on the computer device, 2D coordinates of alocation; entering additional data to the location through a viewer andadding the additional data into the secure database; and translating, bythe application executing on a computer device, the 2D coordinates into3D coordinates and adding the additional data into the 3D CAD model atthe corresponding 2D coordinates of the location.
 4. The method ofmanaging proxy objects within CAD models according to claim 3, furthercomprising automatically capturing the additional data from one or moredata transmitting devices and adding the additional data into the securedatabase.
 5. The method of managing proxy objects within CAD modelsaccording to claim 3, wherein the additional data comprises one or moreof: comments, requests for information, documentation of quality,documentation of safety, text, voice messages, photos, images, and linksto other data on the Internet.
 6. The method of managing proxy objectswithin CAD models according to claim 3, wherein the meta data includesmanufacturer specific information along and associated coordinates ofeach of the objects.
 7. The method of managing proxy objects within CADmodels according to claim 1, further comprising saving the displayedimages of the scene with the selected one or more HD objects at each ofthe specific times with an associated timestamp.
 8. The method ofmanaging proxy objects within CAD models according to claim 7, whereinthe saved images comprise photographic images and wherein displaying thephoto realistic rendered images of the scene includes retrieving auser-selected one of the saved photographic images based on theassociated timestamp
 9. The method of managing proxy objects within CADmodels according to claim 1, wherein the proxy object in the scene ofthe 3D CAD model is associated with a plurality of the HD objects. 10.The method of managing proxy objects within CAD models according toclaim 1, including managing the meta data, wherein the 3D specificationis available inside or outside of a CAD model but connected visually tothe selected one or more HD objects.
 11. The method of managing proxyobjects within CAD models according to claim 1, wherein displaying thephoto realistic rendered images of the scene at the specific timesincludes navigating through the displayed images in response to userinput.
 12. The method of managing proxy objects within CAD modelsaccording to claim 1, wherein at least one of the specific times is realtime, and wherein displaying the photo realistic rendered images of thescene at the specific times includes rendering the images in real timeor at another specific time as requested by the user.
 13. A method ofmanaging proxy objects within CAD models comprising: uploading objectsto a computer device having a memory coupled to an accessible securedatabase in which the objects are stored, the objects including proxyobjects and HD objects matched by meta data attached thereto; placing atleast one the proxy objects in a scene of a 3D CAD model; matching, byan application executing on the computer device, the proxy object in thescene of the 3D CAD model to one or more HD Objects using a productcategory hierarchy; replacing the proxy object in the scene of the 3DCAD model with the matched one or more HD objects; and displaying photorealistic rendered images of the scene at specific times, wherein theproxy object in the scene of the 3D CAD model has been swapped with theselected one or more HD Objects within the scene.
 14. The method ofmanaging proxy objects within CAD models according to claim 12, furthercomprising placing, by the application executing on the computer device,a selected 3D specification icon within the 3D CAD model for a user toaccess via a browser or a mobile app.
 15. The method of managing proxyobjects within CAD models according to claim 12, further comprisingsaving the displayed images of the scene with the selected one or moreHD objects at each of the specific times with an associated timestamp.16. The method of managing proxy objects within CAD models according toclaim 15, wherein the saved images comprise photographic images andwherein displaying the photo realistic rendered images of the sceneincludes retrieving a user-selected one of the saved photographic imagesbased on the associated timestamp.
 17. The method of managing proxyobjects within CAD models according to claim 15, wherein displaying thephoto realistic rendered images of the scene at the specific timesincludes navigating through the displayed images in response to userinput.
 18. The method of managing proxy objects within CAD modelsaccording to claim 15, wherein at least one of the specific times isreal time, and wherein displaying the photo realistic rendered images ofthe scene at the specific times includes rendering the images in realtime or at another specific time as requested by the user.
 19. A systemcomprising: an accessible secure database in which objects are stored,the objects including proxy objects and HD objects matched by meta dataattached thereto; a computer device having a memory coupled to thedatabase, the memory storing computer-executable instructions that, whenexecuted by the computer, configure the system for: placing at least onethe proxy objects in a scene of a 3D CAD model; matching the proxyobject in the scene of the 3D CAD model to at least one of the HDobjects using a product category hierarchy; replacing the proxy objectin the scene of the 3D CAD model with the matched HD object; displayingphoto realistic rendered images of the scene at specific times, whereinthe proxy object in the scene of the 3D CAD model has been swapped withthe matched HD object within the scene; and placing a selected 3Dspecification icon within the 3D CAD model for a user to access via abrowser or a mobile app.
 20. The system according to claim 19, whereinat least one of the specific times is real time, and wherein displayingthe photo realistic rendered images of the scene at the specific timesincludes rendering the images in real time or at another specific timeas requested by the user.